<!DOCTYPE html>
<!-- Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
<html lang="en">
<head>
  <meta charset="utf-8" />
  <title>Vespa Test Framework Tutorial</title>
# Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
  <style type="text/css">

    body {
        margin: 3em;
    }

    body, p, div, div, span, dl, li, pre {
        font-size: 16px;
        line-height: 22px;
    }

    header h1 {
        font-size: 54px;
        line-height: 1;
    }

    section {
        padding-top: 60px;
    }

    ol.linenums {
        padding-left: 20px;
    }

    pre, pre * {
        font-size: 14px !important;
    }

    li.L0, li.L1, li.L2, li.L3, li.L4, li.L5, li.L6, li.L7, li.L8, li.L9
    {
        color: #bbb;
        list-style-type: decimal;
    }

    h2 {
        padding-top: 20px;
        padding-bottom: 10px;
    }
    
    pre.output {
        border: 1px solid #888;
    }

    code {
        color: #444;
        border: 0;
        background-color: transparent;
    }

  </style>
</head>
<body>

<header>
<h1>Vespa Test Framework Tutorial</h1>
</header>

<section>
<div class="page-header">
<h1>Introduction</h1>
</div>

<p>
The Vespa test framework helps you write small applications to test
C++ code. All interaction with the test framework is done with the use
of macros.
</p>

<p>
The minimal test application looks like this:
</p>

<div class="example" id="minimal">
<h2>minimal_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;

TEST_MAIN() {}
</pre>
<pre class="output">
minimal_test.cpp: info:  running test suite 'minimal_test.cpp'
minimal_test.cpp: info:  summary --- 0 check(s) passed --- 0 check(s) failed
minimal_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>

<p>
The runnable application itself is called a <strong>test
suite</strong> and inherits its name from the cpp file containing the
TEST_MAIN macro. Each individual verification of some value is called
a <strong>check</strong>. Checks can be put anywhere, but it is highly
recommended that you put them inside <strong>tests</strong>. Tests are
created by a family of macros. Another macro (TEST_RUN_ALL) is used to
execute them.
</p>

<p>
Example with two tests, each containing a single check:
</p>

<div class="example" id="simple">
<h2>simple_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;

TEST(&quot;require something&quot;) {
    EXPECT_TRUE(true);
}

TEST(&quot;require something else&quot;) {
    EXPECT_TRUE(true);
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
simple_test.cpp: info:  running test suite 'simple_test.cpp'
simple_test.cpp: info:  status_for_test 'require something': PASS
simple_test.cpp: info:  status_for_test 'require something else': PASS
simple_test.cpp: info:  test summary --- 2 test(s) passed --- 0 test(s) failed
simple_test.cpp: info:  imported 2 passed check(s) from 1 thread(s)
simple_test.cpp: info:  summary --- 2 check(s) passed --- 0 check(s) failed
simple_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
</section>


<section>
<div class="page-header">
<h1>Checks</h1>
</div>

<p>
All checks are available in two variants. Those with the
<strong>EXPECT_</strong> prefix allow execution to continue even if a
check fails. Those with the <strong>ASSERT_</strong> prefix will
terminate execution of the current test if it fails.
</p>

<div class="example" id="checks">
<h2>checks_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;
#include &lt;stdexcept&gt;

void willThrow() {
    throw std::runtime_error(&quot;This failed&quot;);
}

TEST(&quot;require that checks work&quot;) {
    EXPECT_TRUE(true);
    EXPECT_FALSE(false);
    EXPECT_EQUAL(3, 3);
    EXPECT_NOT_EQUAL(3, 4);
    EXPECT_APPROX(3.0, 3.1, 0.2);
    EXPECT_NOT_APPROX(3.0, 3.5, 0.2);
    EXPECT_LESS(3, 4);
    EXPECT_LESS_EQUAL(3, 3);
    EXPECT_GREATER(4, 3);
    EXPECT_GREATER_EQUAL(4, 4);
    EXPECT_EXCEPTION(willThrow(), std::runtime_error, &quot;fail&quot;);
}

TEST(&quot;this test will fail&quot;) {
    EXPECT_EQUAL(3, 4);
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
checks_test.cpp: info:  running test suite 'checks_test.cpp'
checks_test.cpp: info:  status_for_test 'require that checks work': PASS
checks_test.cpp: ERROR: check failure #1: '3 == 4' in thread '0(1)' (checks_test.cpp:24)
    STATE[0]: 'this test will fail' (checks_test.cpp:23)
lhs: 3
rhs: 4
checks_test.cpp: ERROR: status_for_test 'this test will fail': FAIL
checks_test.cpp: info:  test summary --- 1 test(s) passed --- 1 test(s) failed
checks_test.cpp: info:  imported 11 passed check(s) from 1 thread(s)
checks_test.cpp: info:  summary --- 11 check(s) passed --- 1 check(s) failed
checks_test.cpp: info:  CONCLUSION: FAIL
</pre>
</div>

<p>
Checks involving comparison of values typically use == and &lt;
operators to compare values. Also; in order to be part of a comparison
check, the value must support the &lt;&lt; operator to print the value
to a string stream in case the check fails.
</p>
</section>


<section>
<div class="page-header">
<h1>Test Fixtures</h1>
</div>
<p>
Sometimes multiple tests wish to use the same predefined state as a
starting point. This state is called a test fixture. Test fixtures are
untangled from the actual tests and construction/destruction is used
to handle their lifetime. When thinking of what can be used as a
fixture, think of what can be put after <strong>new</strong> to create
an object.
</p>

<ul>
<li>A single test can have multiple test fixtures.</li>
<li>The number of <strong>F</strong>s in the test macro denotes the number of fixtures.</li>
<li>Inside the test, fixtures are available as <strong>f1</strong>, <strong>f2</strong> and so forth.</li>
<li>Test fixtures can be parameterized with constructor parameters.</li>
<li>Checks can be performed inside test fixture constructors.</li>
<li>A test fixture constructor can take other test fixtures as input.</li>
</ul>

<div class="example" id="fixtures">
<h2>fixtures_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;

struct Fixture {
    int value;
    Fixture() : value(5) {}
};

TEST_F(&quot;basic fixture&quot;, Fixture) {
    EXPECT_EQUAL(5, f1.value);
}

TEST_FFF(&quot;fancy fixtures&quot;, size_t(10), int(5), std::vector&lt;int&gt;(f1, f2)) {
    EXPECT_EQUAL(10u, f1);
    EXPECT_EQUAL(5, f2);
    ASSERT_EQUAL(10u, f3.size());
    EXPECT_EQUAL(5, f3[7]);
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
fixtures_test.cpp: info:  running test suite 'fixtures_test.cpp'
fixtures_test.cpp: info:  status_for_test 'basic fixture': PASS
fixtures_test.cpp: info:  status_for_test 'fancy fixtures': PASS
fixtures_test.cpp: info:  test summary --- 2 test(s) passed --- 0 test(s) failed
fixtures_test.cpp: info:  imported 5 passed check(s) from 1 thread(s)
fixtures_test.cpp: info:  summary --- 5 check(s) passed --- 0 check(s) failed
fixtures_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
</section>


<section>
<div class="page-header">
<h1>Multi-Threaded Tests</h1>
</div>
<p>
One of the most novel features of the test framework is the ability to
write multi-threaded tests. Multi-threaded tests are created by using
test macros containing <strong>_MT</strong> and supplying a thread
count. All threads will execute the block of code encapsulated by the
test. The test fixtures are shared among all threads. In addition,
each thread has a variable named <strong>num_threads</strong>
containing the total number of threads running the test and a variable
named <strong>thread_id</strong> identifying the thread.
</p>

<p>
The <strong>TEST_BARRIER()</strong> macro can be used inside the test
to synchronize the threads. The macro will block execution of each
thread invoking it until all threads have invoked it.
</p>

<div class="example" id="threads">
<h2>threads_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;

TEST_MT_F(&quot;multiple threads&quot;, 2, std::vector&lt;size_t&gt;(num_threads)) {
    ASSERT_EQUAL(num_threads, f1.size());
    f1[thread_id] = thread_id;
    TEST_BARRIER();
    if (thread_id == 0) {
        TEST_TRACE();
        EXPECT_EQUAL(1u, f1[1]);
    } else {
        TEST_TRACE();
        EXPECT_EQUAL(0u, f1[0]);
    }
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
threads_test.cpp: info:  running test suite 'threads_test.cpp'
threads_test.cpp: info:  trace: thread '0(2)' (threads_test.cpp:9)
threads_test.cpp: info:  trace: thread '1(2)' (threads_test.cpp:12)
threads_test.cpp: info:  status_for_test 'multiple threads': PASS
threads_test.cpp: info:  test summary --- 1 test(s) passed --- 0 test(s) failed
threads_test.cpp: info:  imported 4 passed check(s) from 2 thread(s)
threads_test.cpp: info:  summary --- 4 check(s) passed --- 0 check(s) failed
threads_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
</section>


<section>
<div class="page-header">
<h1>Real World Examples</h1>
</div>
<div class="example" id="../box">
<h2>box_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;
#include &lt;vespa/vespalib/util/box.h&gt;

using namespace vespalib;

void checkValues(const std::vector&lt;int&gt; &amp;values, size_t n) {
    ASSERT_EQUAL(n, values.size());
    for (size_t i = 0; i &lt; n; ++i) {
        EXPECT_EQUAL(int(10 + (10 * i)), values[i]);
    }
}

TEST(&quot;require that boxes can be created and converted to vector&quot;) {
    Box&lt;int&gt; box;
    box.add(10).add(20).add(30);
    checkValues(box, 3);
}

TEST(&quot;require that boxes can be created in place&quot;) {
    checkValues(Box&lt;int&gt;().add(10).add(20).add(30), 3);
}

TEST(&quot;require that make_box works&quot;) {
    checkValues(make_box(10), 1);
    checkValues(make_box(10, 20), 2);
    checkValues(make_box(10, 20, 30), 3);
    checkValues(make_box(10, 20, 30, 40), 4);
    checkValues(make_box(10, 20, 30, 40, 50), 5);
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
box_test.cpp: info:  running test suite 'box_test.cpp'
box_test.cpp: info:  status_for_test 'require that boxes can be created and converted to vector': PASS
box_test.cpp: info:  status_for_test 'require that boxes can be created in place': PASS
box_test.cpp: info:  status_for_test 'require that make_box works': PASS
box_test.cpp: info:  test summary --- 3 test(s) passed --- 0 test(s) failed
box_test.cpp: info:  imported 28 passed check(s) from 1 thread(s)
box_test.cpp: info:  summary --- 28 check(s) passed --- 0 check(s) failed
box_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
<div class="example" id="../barrier">
<h2>barrier_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;
#include &lt;vespa/vespalib/util/barrier.h&gt;

using namespace vespalib;

struct Fixture {
    Barrier barrier;
    CountDownLatch latch;
    Fixture(size_t n) : barrier(n), latch(n) {}
};

TEST_MT_F(&quot;require that barriers are satisfied by the appropriate number of threads&quot;, 3, Fixture(num_threads)) {
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        EXPECT_TRUE(f.barrier.await());
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_TRUE(f1.barrier.await());
        f1.latch.countDown();
    }
}

TEST_MT_F(&quot;require that barriers can be used multiple times&quot;, 3, Fixture(num_threads)) {
    EXPECT_TRUE(f1.barrier.await());
    EXPECT_TRUE(f1.barrier.await());
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        EXPECT_TRUE(f.barrier.await());
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_TRUE(f1.barrier.await());
        f1.latch.countDown();
    }
}

TEST_MT_F(&quot;require that barriers can be broken&quot;, 3, Fixture(num_threads)) {
    EXPECT_TRUE(f1.barrier.await());
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        f1.barrier.destroy();
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_FALSE(f1.barrier.await());
        f1.latch.countDown();
    }
    EXPECT_FALSE(f1.barrier.await());
}

TEST_MT_F(&quot;require that barriers cannot be retroactively broken&quot;, 100, Barrier(num_threads)) {
    EXPECT_TRUE(f1.await());
    f1.destroy();
    EXPECT_FALSE(f1.await());
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
barrier_test.cpp: info:  running test suite 'barrier_test.cpp'
barrier_test.cpp: info:  status_for_test 'require that barriers are satisfied by the appropriate number of threads': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers can be used multiple times': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers can be broken': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers cannot be retroactively broken': PASS
barrier_test.cpp: info:  test summary --- 4 test(s) passed --- 0 test(s) failed
barrier_test.cpp: info:  imported 226 passed check(s) from 100 thread(s)
barrier_test.cpp: info:  summary --- 226 check(s) passed --- 0 check(s) failed
barrier_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
<div class="example" id="../dual_merge_director">
<h2>dual_merge_director_test.cpp</h2>
<pre class="prettyprint linenums">
// Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include &lt;vespa/vespalib/testkit/test_kit.h&gt;
#include &lt;vespa/vespalib/util/dual_merge_director.h&gt;

using namespace vespalib;

struct MySource : public DualMergeDirector::Source {

    bool typeA;
    std::string data;
    std::string diff;

    MySource(bool a, size_t num_sources, size_t source_id)
        : typeA(a),
          data(num_sources, '0'),
          diff(num_sources, '5')
    {
        if (source_id &lt; num_sources) {
            data[source_id] = '1';
            diff[source_id] = '6';
        }
    }
    virtual void merge(Source &amp;mt) {
        MySource &amp;rhs = static_cast&lt;MySource&amp;&gt;(mt);
        ASSERT_EQUAL(typeA, rhs.typeA);
        ASSERT_EQUAL(data.size(), rhs.data.size());
        for (size_t i = 0; i &lt; data.size(); ++i) {
            int d = (rhs.data[i] - '0');
            data[i] += d;
            diff[i] += d;
            rhs.diff[i] -= d;
        }
    }
    void verifyFinal() const {
        EXPECT_EQUAL(std::string(data.size(), '1'), data);
        EXPECT_EQUAL(std::string(diff.size(), '6'), diff);
    }
    void verifyIntermediate() const {
        EXPECT_EQUAL(std::string(diff.size(), '5'), diff);
    }
};

TEST_MT_F(&quot;require that merging works&quot;, 64, std::unique_ptr&lt;DualMergeDirector&gt;()) {
    for (size_t use_threads = 1; use_threads &lt;= num_threads; ++use_threads) {
        MySource sourceA(true, use_threads, thread_id);
        MySource sourceB(false, use_threads, thread_id);
        if (thread_id == 0) {
            f1.reset(new DualMergeDirector(use_threads));
        }
        TEST_BARRIER();
        if (thread_id &lt; use_threads) {
            f1-&gt;dualMerge(thread_id, sourceA, sourceB);
        }
        TEST_BARRIER();
        if (thread_id == 0) {
            sourceA.verifyFinal();
            sourceB.verifyFinal();
        } else if (thread_id &lt; use_threads) {
            sourceA.verifyIntermediate();
            sourceB.verifyIntermediate();
        }
    }
}

TEST_MAIN() { TEST_RUN_ALL(); }
</pre>
<pre class="output">
dual_merge_director_test.cpp: info:  running test suite 'dual_merge_director_test.cpp'
dual_merge_director_test.cpp: info:  status_for_test 'require that merging works': PASS
dual_merge_director_test.cpp: info:  test summary --- 1 test(s) passed --- 0 test(s) failed
dual_merge_director_test.cpp: info:  imported 12352 passed check(s) from 64 thread(s)
dual_merge_director_test.cpp: info:  summary --- 12352 check(s) passed --- 0 check(s) failed
dual_merge_director_test.cpp: info:  CONCLUSION: PASS
</pre>
</div>
</section>


<section>
<div class="page-header">
<h1>Macro Summary</h1>
</div>

<h2>Overall Execution Macros</h2>
<ul>
<li><code>TEST_MAIN()</code></li>
<li><code>TEST_RUN_ALL()</code></li>
</ul>

<h2>Test Creation Macros</h2>
<ul>
<li><code>TEST(name)</code></li>
<li><code>TEST_F(name, fixture)</code></li>
<li><code>TEST_FF(name, fixture1, fixture2)</code></li>
<li><code>TEST_FFF(name, fixture1, fixture2, fixture3)</code></li>
</ul>

<ul>
<li><code>TEST_MT(name, threads)</code></li>
<li><code>TEST_MT_F(name, threads, fixture)</code></li>
<li><code>TEST_MT_FF(name, threads, fixture1, fixture2)</code></li>
<li><code>TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)</code></li>
</ul>

<ul>
<li><code>IGNORE_TEST(name)</code></li>
<li><code>IGNORE_TEST_F(name, fixture)</code></li>
<li><code>IGNORE_TEST_FF(name, fixture1, fixture2)</code></li>
<li><code>IGNORE_TEST_FFF(name, fixture1, fixture2, fixture3)</code></li>
</ul>

<ul>
<li><code>IGNORE_TEST_MT(name, threads)</code></li>
<li><code>IGNORE_TEST_MT_F(name, threads, fixture)</code></li>
<li><code>IGNORE_TEST_MT_FF(name, threads, fixture1, fixture2)</code></li>
<li><code>IGNORE_TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)</code></li>
</ul>

<h2>Check Macros</h2>
<ul>
<li><code>ASSERT_TRUE(rc)</code></li>
<li><code>ASSERT_FALSE(rc)</code></li>
<li><code>ASSERT_EQUAL(a, b)</code></li>
<li><code>ASSERT_NOT_EQUAL(a, b)</code></li>
<li><code>ASSERT_APPROX(a, b, eps)</code></li>
<li><code>ASSERT_NOT_APPROX(a, b, eps)</code></li>
<li><code>ASSERT_LESS(a, b)</code></li>
<li><code>ASSERT_LESS_EQUAL(a, b)</code></li>
<li><code>ASSERT_GREATER(a, b)</code></li>
<li><code>ASSERT_GREATER_EQUAL(a, b)</code></li>
<li><code>ASSERT_EXCEPTION(statement, exception_type, msg_substr)</code></li>
<li><code>TEST_FATAL(msg)</code></li>
</ul>

<ul>
<li><code>EXPECT_TRUE(rc)</code></li>
<li><code>EXPECT_FALSE(rc)</code></li>
<li><code>EXPECT_EQUAL(a, b)</code></li>
<li><code>EXPECT_NOT_EQUAL(a, b)</code></li>
<li><code>EXPECT_APPROX(a, b, eps)</code></li>
<li><code>EXPECT_NOT_APPROX(a, b, eps)</code></li>
<li><code>EXPECT_LESS(a, b)</code></li>
<li><code>EXPECT_LESS_EQUAL(a, b)</code></li>
<li><code>EXPECT_GREATER(a, b)</code></li>
<li><code>EXPECT_GREATER_EQUAL(a, b)</code></li>
<li><code>EXPECT_EXCEPTION(statement, exception_type, msg_substr)</code></li>
<li><code>TEST_ERROR(msg)</code></li>
</ul>

<h2>Thread Macros</h2>
<ul>
<li><code>TEST_BARRIER()</code></li>
</ul>

<h2>State Tracking Macros</h2>
<ul>
<li><code>TEST_DO(doit)</code></li>
<li><code>TEST_STATE(msg)</code></li>
</ul>

<h2>Macros of Limited Use</h2>
<ul>
<li><code>TEST_TRACE()</code></li>
<li><code>TEST_FLUSH()</code></li>
<li><code>TEST_THREAD(name)</code></li>
<li><code>TEST_DEBUG(lhsFile, rhsFile)</code></li>
<li><code>TEST_MAIN_WITH_PROCESS_PROXY()</code></li>
</ul>
</section>

<script type="text/javascript">
$(function(){
    window.prettyPrint && prettyPrint();
});

</script>

</body>
</html>
